Skip to content

Add posterior CRPS, Brier scores and finite-action risk metrics - #9

Merged
robinthibaut merged 1 commit into
mainfrom
robin/skbel-posterior-metrics
Sep 10, 2026
Merged

robinthibaut merged 1 commit into
mainfrom
robin/skbel-posterior-metrics

Conversation

@robinthibaut

Copy link
Copy Markdown
Owner

Summary

Add a reusable skbel.metrics module for:

  • marginal CRPS, with weighted empirical and unweighted iid U estimators;
  • multiclass Brier scores (sum across classes, explicitly documented);
  • posterior expected losses over a finite action set;
  • all exactly tied minimum-risk actions.

Inputs have explicit shape, finite-value and weight contracts with no implicit
broadcasting. Weight normalization rescales before summation to avoid overflow
from large finite weights. Include API notes and a small executable example.

Verification and limits

48 focused tests cover hand-computed and brute-force score comparisons, scaling
invariance, large/tiny/zero weights, invalid inputs, tied actions and ordinary
package imports. Full local upstream tests and Ruff checks pass; repository CI
checks supported Python/OS combinations. No dependency changes.

These are general scoring and terminal-decision arithmetic, not an acquisition
policy or proof of posterior calibration. CRPS is marginal, not a joint score;
the U estimator requires iid unweighted draws; Brier's two-class result is twice
the scalar binary convention. Future-observation simulation/updating and domain
loss choices remain caller responsibilities.

@robinthibaut
robinthibaut merged commit 22b9a73 into main Sep 10, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant